AIR CLI: drop max_retries from the ai_runtime_task payload#5813
Merged
Conversation
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Contributor
Waiting for approvalCould not determine reviewers from git history. Eligible reviewers: Suggestions based on git history. See OWNERS for ownership rules. |
Collaborator
Integration test reportCommit: f17e688
10 interesting tests: 4 SKIP, 3 flaky, 3 RECOVERED
Top 29 slowest tests (at least 2 minutes):
|
riddhibhagwat-db
force-pushed
the
air-max-retries-fix
branch
from
3de55a3 to
f5eb71b
Compare
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
maggiewang-db
left a comment
maggiewang-db
left a comment
There was a problem hiding this comment.
execution retries are driven by the AI Runtime service (AICM) - Is it true?
@ben-hansen-db How is a user supposed to specify the max retry number with the new BYOT?
riddhibhagwat-db
force-pushed
the
air-max-retries-fix
branch
from
f5eb71b to
b25accb
Compare
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
The ai_runtime_task path DOES honor the Jobs task max_retries field —
Jobs performs the retry and each attempt is created as a fresh AI Runtime
(AICM) workload (see ai-training AiTrainingHandlers: "Each Jobs retry
attempt is created as its own AICM workload"). So the retry knob is real.
Send max_retries whenever it is set (including 0), so the user's YAML
value is honored end-to-end:
- max_retries unset -> defaults to 3 (matches Python: sdk/config.py
Field(default=3)).
- max_retries: N>0 -> sent, retry_on_timeout: true.
- max_retries: 0 -> sent explicitly ("max_retries":0), retry_on_timeout
omitted. This diverges from the Python CLI (which drops the field
under `if max_retries > 0`) so that 0 actually means "no retries"
instead of silently falling back to the server default.
Co-authored-by: Isaac
riddhibhagwat-db
force-pushed
the
air-max-retries-fix
branch
from
b25accb to
f17e688
Compare
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
vinchenzo-db
reviewed
Jul 8, 2026
vinchenzo-db
left a comment
vinchenzo-db
left a comment
There was a problem hiding this comment.
Seems pr desc may be out of date. Also @maggiewang-db I believe Jobs drives the retry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Send max_retries on the ai_runtime_task submit payload whenever it is set, so the user's YAML value is honored end-to-end:
max_retries is always marshaled (no omitempty) so an explicit 0 reaches the server. retry_on_timeout keeps omitempty and is set only when max_retries > 0, so the wire form matches the Python CLI (which never emits a bare retry_on_timeout: false). The YAML field and its >= 0 validation are unchanged.
Why
On the ai_runtime_task path, retries are honored via the Jobs task max_retries field: Jobs performs the retry, and each attempt is created as its own AIR workload.
The bug was that max_retries was previously sent unconditionally including 0, but 0 was being dropped/ignored so that setting max_retries: 0 to disable retries silently fell back to the server default and the workload still retried. Sending 0 explicitly makes "no retries" actually take effect.
This intentionally diverges from the Python CLI's native branch by one case: Python only adds the field under if max_retries > 0, so Python's max_retries: 0 silently reverts to the server default. We send 0 so the user's intent is honored.
Tests